Evaluating the Inference Mechanism of Adaptive Learning Systems

The evaluation of user modeling systems is an important though often neglected area. Evaluating the inference of user properties can help to identify failures in the user model. In this paper we propose two methods to assess the accuracy of the user model. The assumptions about the user might either be compared to an external test, or might be used to predict the users’ behavior. Two studies with five adaptive learning courses demonstrate the usefulness of the approach. 1 Evaluation of Adaptive Systems Empirical evaluations of adaptive systems are rare [11]—e.g., only a quarter of the articles published in User Modeling and User Adapted Interaction report significant empirical evaluations. Many of them include a simple evaluation study with small sample sizes (often N = 1) and without any statistical method. Several reasons for this absence of significant studies have been identified (e.g., [6,7]). A systematic overview of current evaluation studies can be found in EASy-D, an online database of adaptive systems and related empirical studies1. Recently, we proposed an evaluation framework that supports and encourages researchers to evaluate their adaptive system by separating different evaluation steps [11]: evaluation of the input data, evaluation of the inference mechanism, evaluation of the adaptation decision, and evaluation of the interaction. In this paper we focus on the second step. The inference mechanism is the crucial part of many adaptive systems. We propose two methods that test the accuracy of a user model – by comparing its assumptions to an external test, and – by comparing its assumptions to the actually displayed behavior of the learners. These two methods are applied to an adaptive learning system, called the HTML-Tutor, to demonstrate the usefulness of the approach. ?? c © by Springer-Verlag 1 http://www.softwareevaluation.de 2 Adaptive Learning Systems built with NetCoach Adaptive learning systems adapt their behavior to individual learner properties such as the user’s current knowledge. Opposed to static learning systems that present the same material to every user in the same way and order, adaptive systems consider individual differences in terms of knowledge, experience, preferences, or learning objectives [3] and thus promise to improve the teaching process [8]. NetCoach2 is an authoring system that enables authors to develop adaptive webbased learning courses without being required to program source code [10]. 2.1 Course Structure and Adaptivity All NetCoach courses are based on the same structure. Similar to chapters and subchapters in a book, the learning material (i.e., pages with texts, images, animations) is stored in a hierarchical tree-structure of concepts. Learners may navigate through this structure freely. However, the course adapts to each learner individually by suggesting concepts that are suitable to work on next (adaptive curriculum sequencing) and by annotating the links to other concepts (adaptive link annotation). This functionality is based on two kinds of data: concept relations and test sets (also called test groups) that check the learners knowledge about a concept. Authors may define two kinds of relations between concepts as regards contents. First, a concept might be prerequisite to another, i.e., this concept should be learned before the second concept is presented. Second, a concept might infer another concept, i.e., the fact that the learner knows concept A implies that she also knows concept B. The crucial part of NetCoach to assess the learner’s knowledge are the so called test sets. A test set consists of a set of weighted test items that are related to a concept. There are forced choice, multiple choice, gap filling, and ranking items. All of them are evaluated online automatically. Users receive points for answering a test item correctly. Mistakes result in a reduction of points. Items are presented randomly (not yet presented items and incorrectly answered items are preferred) until the learner reaches a critical value. Only then the related concept is assumed to be learned completely. 2.2 Inference Mechanism Adapting to the learner’s current knowledge is one of the most important features of NetCoach. If the user completed a test set successfully (either during an introductory test before the content of the concept has been presented or afterwards in a post test) the concept is marked as solved and further inferences about the user’s knowledge are drawn based on the inference relations between concepts. NetCoach summarizes the learner’s current knowledge by assigning one of five states to each concept. Table 1 lists the states and describes the conditions of assignment. The current configuration of states is called a user’s learning state. As it is computed on the fly for each user individually, the learning state models the idiosyncratic learning process during the interaction. 2 http://art.ph-freiburg.de Table 1. Possible states of a concept with a test set. The states are computed individually during interaction in dependence of the user’s behavior. state condition annotation not ready there are prerequisites for a concept (e.g., concept A has to be learned before concept B) that are not fulfilled red ball suggested all prerequisites are fulfilled green ball solved the learner completed the test set of this concept successfully grey ball with tick inferred the learner solved a more advanced concept first and thus the current concept is inferred to be already learned as well. orange ball with tick known the learner marked the concept as known without solving the test set crossed orange ball We argue that it is insufficient to assess the current knowledge by looking at the visited pages as most adaptive systems do (e.g., AHA [5] or Interbook [4]). An explicit assessment with sets of test items provides a much more reliable user model and might thus support better and adequate adaptations of the interface. 2.3 Adaptation Decision Based on this inferred individual learning state NetCoach adapts its interface in two ways. First, links to other concepts are annotated with colored bullets that correspond to the learning state (adaptive link annotation). Table 1 gives an example of the default color configuration, however, authors are free to predefine other colors for each state. Second, NetCoach suggests a concept that should be learned next and gives warnings if the learner visits a concept with the state not ready (adaptive curriculum sequencing). Note, that for these adaptation techniques the quality of the learning state assessment is crucial. The adaptation decision will only work if the underlying assumptions about the learner are correct. Thus, the two methods to assess the accuracy of the user model are an important prerequisite to a successful adaptation. 3 Comparison of User Model to an External Test The first proposed method to test the accuracy of a user model is to compare the assumptions in the user model to an external test. For instance, the assumption of a product recommendation system [2] about the most preferred product could be tested by actually letting the customer choose between several products. The user model of a system that adapts to the user’s keyboard skills [9] might be tested by assessing the skills externally with a valid diagnostic instrument for motor disabilities. In an adaptive learning system it is possible to compare the assumed knowledge of the learner to the results of an external knowledge test that is known to have external validity. We evaluated the congruence of the user models in the HTML-Tutor and an extended external assessment. The HTML-Tutor is a NetCoach course that introduces to HTML and publishing on the internet. It consists of 138 concepts, 48 test sets, and 125 test items.